Learning to Predict Image-based Rendering Artifacts with Respect to a Hidden Reference Image

Image metrics predict the perceived per-pixel difference between a reference image and its degraded (e. g., re-rendered) version. In several important applications, the reference image is not available and image metrics cannot be applied. We devise a neural network architecture and training procedure that allows predicting the MSE, SSIM or VGG16 image difference from the distorted image alone while the reference is not observed. This is enabled by two insights: The first is to inject sufficiently many un-distorted natural image patches, which can be found in arbitrary amounts and are known to have no perceivable difference to themselves. This avoids false positives. The second is to balance the learning, where it is carefully made sure that all image errors are equally likely, avoiding false negatives. Surprisingly, we observe, that the resulting no-reference metric, subjectively, can even perform better than the reference-based one, as it had to become robust against mis-alignments. We evaluate the effectiveness of our approach in an image-based rendering context, both quantitatively and qualitatively. Finally, we demonstrate two applications which reduce light field capture time and provide guidance for interactive depth adjustment.Comment: 13 pages, 11 figure

Guided Optimization Framework for the Fusion of Time of Flight with Stereo Depth

The fusion of depth acquired actively with the depth estimated passively proved its significance as animprovement strategy for gaining depth. This combination allows us to benefit from two sources of modalities suchthat they complement each other. To fuse two sensor data into a more accurate depth map, we must consider thelimitations of active sensing such as low lateral resolution while combining it with a passive depth map. In this paper,we present a novel approach for the fusion of active Time of Flight (ToF) depth and passive stereo depth in an accurateway. We propose a multimodal sensor fusion strategy that is based on a weighted energy optimization problem. Theweights are generated as a result of combining the edge information from a texture map as well as active and passivedepth maps. The objective evaluation of our fusion algorithm shows an improved accuracy of the generated depthmap in comparison with the depth map of every single modality as well as with the results of other fusion methods.Additionally, a visual comparison of our result shows a better recovery on the edges considering the wrong depthvalues estimated in passive stereo. Moreover, the left and right consistency check on the result illustrates the abilityof our approach to consistently fusing sensors

Simulative buffer analysis of local image processing algorithms described by windowed synchronous data flow

{ haubelt,teich} @ cs.fau.de Abstract- Embedded real-time image processing applications working on large images have to process and store huge amounts of data. Consequently the organization of the memory buffers and the precise determination of the required buffer sizes are critical steps for efficient system implementation. In this paper, we propose a new method, that permits the analysis to be performed automatically for local image processing algorithms. The latter ones are specified by help of the Windowed Synchronous Data Flow (WSDF) model, a multi-dimensional model of computation which has been especially designed to represent local image processing algorithms. This paper introduces a corresponding buffer organization leading to solutions comparable to hand-built designs concerning the required memory. Special care is taken, so that also large problems in terms of the image size can be analyzed. The applicability of our approach is demonstrated by help of a JPEG2000 decoder model

A Generalized Static Data Flow Clustering Algorithm for MPSoC Scheduling of Multimedia Applications ABSTRACT

In this paper, we propose a generalized clustering approach for static data flow subgraphs mapped onto individual processors in Multi-Processor System on Chips (MPSoCs). The goal of clustering is to replace the static data flow subgraph by a single dynamic data flow actor such that the global performance in terms of latency and throughput is optimized. Through our proposed clustering approach, the scheduling of connected static data flow subgraphs can be coordinated with enclosing system representations in a way that systematically exploits the predictability and efficiency of the static data flow model. Thus, the advantages of static data flow subsystems can be exploited in the context of overall system representations that are based on more general models of computation. At the same time, our approach goes significantly beyond previous approaches to synchronous data flow clustering by providing a quasi-static —asopposedto purely-static — scheduling interface between clustered subgraphs and the enclosing systems. This greatly enhances the power of our techniques in terms of avoiding deadlock, increasing the design space for clustering, and providing for integration with more general models of computation. We show benefits of up to 95 % performance improvement for real world examples

Pixel-Wise Confidences for Stereo Disparities Using Recurrent Neural Networks

One of the inherent problems with stereo disparity estimation algorithms is the lack of reliability information for the computed disparities. As a consequence, errors from the initial disparity maps are propagated to the following processing steps such as view rendering. Nowadays, confidence measures belong to the most popular techniques because of their capability to detect disparity outliers. Recently, convolutional neural network based confidence measures achieved best results by directly processing initial disparity maps. In contrast to existing convolutional neural network based methods, we propose a novel recurrent neural network architecture to compute confidences for different stereo matching algorithms. To maintain a low complexity the confidence for a given pixel is purely computed from its associated matching costs without considering any additional neighbouring pixels. As compared to the state-of-the-art confidence prediction methods leveraging convolutional neural networks, the proposed network is simpler and smaller in terms of size (reduction of the number of trainable parameters by almost 3-4 orders of magnitude). Moreover, the experimental results on three well-known datasets as well as with two popular stereo algorithms clearly highlight that the proposed approach outperforms state-of-the-art confidence estimation techniques

Compressed Data Stream Transmission using Rate Control

Logging the fill level of a virtual (encoder output) buffer at both the encoding and decoding sides is used to provide a common basis for adjusting the rate control parameter so that the latter does not have to be explicitly signaled to the decoder